Highly Optimized Tolerance: Robustness and Power Laws in Complex Systems 
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We introduce highly optimized tolerance (HOT), a mechanism that connects evolving structure and power 
laws in interconnected systems. HOT systems arise, e.g., in biology and engineering, where design and 
evolution create complex systems sharing common features, including (1) high efficiency, performance, 
and robustness to designed-for uncertainties, (2) hypersensitivity to design flaws and unanticipated 
perturbations, (3) nongeneric, specialized, structured configurations, and (4) power laws. We introduce 
HOT states in the context of percolation, and contrast properties of the high density HOT states with 
random configurations near the critical point. While both cases exhibit power laws, only HOT states 
display properties (1-3) associated with design and evolution. 

PACS numbers: 05.40. +j, 64.60.Ht, 64.60. Lx, 87.22. As, 89.20.+a 



Evolution from primitive isolation to more dense inter- 
connection is an important strategy for both biological |^] 
and technological systems as they progress towards in- 
creasing robustness and higher performance. However, in- 
terconnections also make systems more vulnerable to catas- 
trophic breakdowns associated with cascading failures initi- 
ated by seemingly innocuous local events. Recently a great 
deal of attention has been given to the fact that many com- 
plex systems share a common statistical attribute: the dis- 
tributions of sizes of events satisfy power laws j^] . Examples 
' include the probability distributions describing the num- 
ber of fatalities and/or economic losses due to earthquakes, 
' hurricanes, floods, epidemics, and social conflicts, cus- 
\ tomers affected by power outages [D , delays associated with 
• traffic jams |^], and many quantities on the internet Q. 
\ In this letter we introduce a mechanism for power laws 
. which is relevant for systems which are optimized, either 
' by design or natural selection, for high output in the pres- 
, ence of some external hazard. Optimization causes systems 
' to evolve away from generic states towards rare, special- 
, ized configurations. Interestingly, we find that tradeoffs 
between maximizing yield and minimizing risk quite gener- 
ically leads to heavy tails (power laws) in the distribution 
of failure events. We refer to our mechanism as highly op- 
timized tolerance (HOT), suggesting systems designed for 
high performance in an uncertain environment, and oper- 
ating at densities well above the standard critical point. 
However, along with the high performance comes vulnera- 
bility and brittleness with respect to design flaws and unan- 
ticipated changes in the external conditions. 

Our mechanism provides a sharp contrast with the widely 
popularized alternative scenario in which open systems 
evolve to a critical or bifurcation point. In that picture, 
systems are said to be at the "edge of chaos" or in a 
self-organized critical (SOC) state In model systems 
the internal dynamics lead a key macroscopic control pa- 
rameter or density to converge to the critical point. The 



system is otherwise free to explore a wide variety of micro- 
scopic configurations which are consistent with the specified 
density, and power laws and self-similarity arise as familiar 
hallmarks of criticality. 
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FIG. 1. Sample configurations for (a) the random case near 
Pc (b) a HOT grid, and HOT states obtained by evolution at 
(c) optimal yield, and (d) a somewhat lower density. Unoccu- 
pied sites are black, and clusters are grey, where darker shades 
indicate larger clusters. 

We focus on a very simple setting, two-dimensional site 
percolation on an A'' x TV square lattice. We use A^ = 32 
throughout for the numerical examples, so that features of 
specific configurations in Fig. 1 are easily visualized. We 
have verified numerically and in some cases analytically jl^ 
that the power laws extend to large A^, where the statis- 
tics become smoother and the transitions sharper. In the 
random case (i.e. no design) sites are independently occu- 
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pied with probability p and vacant with probabihty (1 — p), 
so that for a given density p = p all configurations are 
equally likely. In contrast, design implies a selection of spe- 
cial configurations, in our case associated with optimization 
for yield in the presence of external risk. 

In the standard forest analogy, occupied sites correspond 
to trees, and risk is associated with fires. The yield Y is 
defined to be the average density of trees left unburnt af- 
ter a spark hits. If a spark hits an unoccupied site, nothing 
burns. When the spark hits an occupied site the fire spreads 
throughout the associated cluster, defined to be the con- 
nected set of c nearest neighbor occupied sites. Let /(c) 
denote the distribution of events of size c, and let F(c) de- 
note the cumulative distribution of events greater than or 
equal to c. The yield is then Y{p) — p— < / > where the 
average < / > is computed with respect to both the ensem- 
bles of configurations and the spatial distribution P{i,j) of 
sparks. By translation invariance, results for the random 
case are independent of the distribution of sparks, while 
P{i,j) is a central ingredient for the design of tolerant con- 
figurations. We assume P{i,j) is given, e.g., in terms of the 
past history of events. HOT states arise when we optimize 
the yield Y. 

In Fig. 2a we plot yield Y as function of the initial density 
p for a variety of different scenarios. The maximum pos- 
sible yield corresponds to the diagonal line: Y = p, which 
is obtained if a vanishing fraction of the sites are burned 
after the spark lands. In the limit ^ oo it is possible 
to attain the maximum yield for the full range of densities. 
The diagonal breaks up into three regimes: a range of iso- 
lated states composed of small, well separated clusters at 
low densities, which terminates in a generic critical point, 
Pc, beyond which maximum yield is obtainable only for a 
measure zero subset of tolerant high density configurations. 

The yield curve for the random case is depicted by the 
dashed line in Fig. 2a, and illustrates the isolated and crit- 
ical regimes. At low densities the results coincide with the 
maximum yield. Near p = pc there is a crossover, and Y{p) 
begins to decrease monotonically with p, approaching zero 
at high density. The crossover becomes sharp as N oo 
and is an immediate consequence of the percolation tran- 
sition, marking the emergence of an infinite cluster when 
p = Pc- In the thermodynamic limit only events involv- 
ing the infinite cluster result in a macroscopic event and 
Y{p) = p — Poc^{p). Here Poo(p) is the percolation order 
parameter, i.e., the probability a given site is in the infinite 
cluster. A typical random configuration at peak yield is il- 
lustrated in Fig. la. The fractal appearance of the clusters 
is a key signature of criticality. 

The goal of design is to push the yield towards the up- 
per bound for densities which exceed the critical point. This 
requires selecting nongeneric (measure zero) configurations, 
which we refer to as tolerant states. We define HOT states 
to be those which specifically optimize yield in the presence 
of a constraint (see Figs. Ib-d). Unlike random configura- 
tions, in tolerant states the connected clusters are regular 
in shape and separated by well defined barriers consisting 
of closed contours of unoccupied sites. A HOT state cor- 



responds to a forest which is densely planted to maximize 
the timber yield, with fire breaks arranged to minimize the 
spread of damage. 

Optimization requires that we specify P{i,j) and any 
applicable constraints. Precise knowledge of the position 
(i, j) of the next spark trivially leads to HOT states where 
that site is vacant, and hence no fire. Alternately, if P{i,j) 
is spatially uniform, HOT states consist of regular cells of 
equal size (in the thermodynamic limit). More interesting 
cases arises when P(i,j) is a nontrivial distribution, with 
regions of high and low probability. For our numerical ex- 
amples we use: 

P(z,j) = P(*)P(j) 
P{x) cx 2-[(™-+(^/^))/^-l' (1) 

where rui = 1, Ci = 0.4, mj = 0.5 and Uj = 0.2. We 
choose the tail of a Gaussian to dramatize that power laws 
emerge through design even when the external distribution 
is far from a power law. Otherwise Eq. (|l|) is chosen some- 
what arbitrarily to avoid artificial symmetries in the HOT 
configurations. 




FIG. 2. Comparison between HOT states and random sys- 
tems at criticality: (a) Yield vs. Density: Y{p), and (b) cumu- 
lative distributions of events F{c) for cases (a)-(d) in Fig. 1. 

Specific constraints make the optimization procedure 
tractable. The HOT configuration illustrated in Fig. lb 
is obtained subject to a constraint in which the lattice is 
fully occupied except for horizontal and vertical lines of va- 
cant sites or "cuts" which divide the system into rectangular 
clusters. In this case it is straightforward to determine the 
constrained, global optimum by searching over the number 
and locations of cuts. Analytical calculations can be made 
in the thermodynamic limit [|ll|. Solutions for Cauchy, ex- 
ponential, and Gaussian distributions P(i,j) in addition to 
Eq. (0) , exhibit power law tails in the distribution of burn 
events, where for a broad class of distributions the scaling 
is asymptotically independent of P(i, j). Numerical results 
for the case of Eq. (P are illustrated in Fig. 2b. The key 
point is that in the tolerant regime power laws events are 
highly generic for a variety of (not necessarily power law) 
input distributions. In HOT states resources (in this case 
the cuts) are concentrated around regions of high P{i,j), 
creating small clusters, while few resources are spent where 
P{i,j) is small, creating large clusters. Since the probabil- 
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ity of events /(c) is the sum of all the P{i,j) in clusters of 
size c, optimizing yield balances cluster size and probability, 
which produces power law tails. 

There are many alternative optimization schemes associ- 
ated with different constraints. Next wc turn to a local and 
incremental algorithm, which is reminiscent of evolution by 
natural selection. We begin with an empty lattice, and add 
grains one at a time to sites which maximize expected yield 
at each step. For asymmetric P{i,j) such as Eq. (|l|) this 
algorithm is deterministic. We obtain a sequence of config- 
urations of monotonically increasing density, which passes 
through the critical density Pc unobstructed. Indeed, Pc 
plays no special role. At much higher densities there is a 
maximum yield point followed by a drop in the yield. The 
yield curve Y{p) is plotted in Fig. 2a for the P{i,j) given 
in Eq. (0). 

A sample HOT configuration generated by this algorithm 
is illustrated in Fig. Ic for a density near the maximum yield 
point in Fig. 2a. This optimization explores only a small 
fraction of the configurations at each density p. Specif- 
ically, (1 — p)N'^ of the ((iJ!^)jv2) possible configurations 
are searched. Nonetheless, yields above 0.9 are obtained 
on a 32 X 32 lattice, and in the thermodynamic limit the 
peak yield approaches the maximum value of unity. While 
the clusters are not perfectly regular, the configuration has 
a clear cellular pattern, consisting of compact regions en- 
closed by well defined barriers. As shown in Fig. 2b, the 
distribution of events F(c) exhibits a power law tail when 
P{i,j) is given by Eq. (|l|). This is the case for a broad class 
of P(i,j), including Gaussian, exponential, and Cauchy. 

Interestingly, in the tolerant regime our algorithm pro- 
duces power law tails for a range of densities below the 
maximum yield, and without ever passing through a state 
that resembles the (fractal) critical state. This is illustrated 
in Figs. Id and 2b where we plot the event size distribution 
F{c) (lower of the "evolved" curves) for a density which 
lies below that associated with the peak yield. Note that 
this configuration has many clusters of unit size c = 1 in 
checkerboard patterns in the region of high P(i,j) in the 
upper left corner. The fact that power laws are not a special 
feature associated with a single density is in sharp contrast 
to a traditional critical phenomena. 

Like criticality, HOT states display certain "universal" 
features, and the scaling properties are determined by lim- 
ited sets of key variables. These variables are different in 
the two cases, but the most interesting properties of HOT 
states are those which even more clearly distinguish toler- 
ance from criticality. In contrast to the fractal percolation 
clusters, regions in the HOT state are both regular and 
structured depending on the P(i,j), and also highly sensi- 
tive to changes in P(i,j). For example, if a configuration 
which is optimized for a Gaussian distribution is subjected 
to a uniform distribution of hits, then the distribution of 
events increases with size: /(c) ^ c ]ri[ |. For the random 
critical case the event size distribution is a priori indepen- 
dent of the spark distribution. 

For a given density the expected event sizes associated 



with HOT states are much smaller than those of random 
configurations. In Fig. 2b the random case exhibits the flat- 
test distribution with events of the largest average size, in 
spite of the fact that it corresponds to the lowest density. 
However, there is a robustness tradeoff which introduces 
new sensitivities in the HOT state which are not present 
in random cases. For example, the HOT state is extremely 
sensitive to design flaws. If an element in the surround- 
ing barrier of vacant sites is absent (that is, occupied by a 
tree) , then fire leaks through the barrier into the surround- 
ing regions. In contrast, in random configurations small 
changes do not alter the distribution of events. Robustness 
to additional uncertainties such as design flaws or multiple 
sparks can be designed for at some cost in yield. A common 
engineering design strategy is to simply back off from the 
peak yield (e.g. Fig. Ic), and consider a conflguration more 
analogous to that illustrated in Fig. Id. 

In summary, the distinguishing features of the HOT state 
include (1) high yields robust to designed- for uncertainty, 
(2) hypersensitivity to design flaws and unanticipated per- 
turbations, (3) stylized and structured conflgurations, and 
(4) power law distributions. Percolation seems to be the 
simplest template for introducing HOT states and contrast- 
ing their properties with criticality. For a more unifled per- 
spective, the random case can be viewed as a very primitive 
design with density as the only design parameter. In this 
case, the critical point coincides with the maximum yield, 
making this a natural alternative to SOC whereby primi- 
tive systems might evolve to criticality. More importantly, 
adding even modest levels of additional design moves yields 
well above the random critical point. While both HOT 
states and critical points exhibit power laws, this is the 
least consequential of the four noted HOT features. 

This simple model is emphatically not meant to real- 
istically represent any speciflc system, and is at best re- 
motely connected with forest management. Nevertheless, 
it is striking how commonly complex systems have all the 
features of the HOT state. We briefly review a few of these 
systems, particularly those previously studied emphasizing 
power laws and criticality, in order to underscore that power 
law statistics alone should not necessarily be interpreted as 
signatures of criticality. For example, in this context, high- 
way traffic is widely studied 0. However, the complete 
highway system is highly structured with throughput dom- 
inated by design, including multiple and specialized lanes 
and ramps, the use of buses and vans, and perhaps most 
importantly, drivers capable of sophisticated active feed- 
back control and collision avoidance. Traffic flow can also 
be hypersensitive to, say, accidents that block lanes. 

Modern computer networks, which exhibit many power 
laws are also highly structured with routers, caches, and 
sophisticated multilayer protocols to provide high through- 
put. The network is robust to moderate variations in traffic, 
or loss of a router or line, but extremely sensitive to bugs 
in network software, underscoring the importance of soft- 
ware reliability. The Ariane 5 crash and Y2K problems are 
a few other examples of our vulnerability in software inten- 
sive systems. The high connectivity and throughput of the 
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electric power network, plus protective relay control, pro- 
vide great robustness to most perturbations, but can also 
lead to large cascading multibillion-doUar failures from ap- 
parently small and innocuous initiating events Q . 

Biological systems show extreme robustness at all levels, 
including cells, organisms, and ecosystems, but also have 
hypersensitivity to the alteration of a single gene or specie. 
Despite possibly fractal signaling and transport structures, 
which are themselves products of design, the overall hi- 
erarchy of organism, organs, cells, organelles, and macro- 
molecules is highly structured. Finally, design plays a sur- 
prisingly large role even in natural disasters such as hur- 
ricanes, earthquakes, floods, and tornadoes. According to 
a recent report , economic losses associated with natural 
disasters are on the rise due to an increased concentration of 
population and infrastructure in high risk areas. Thus the 
authors argue that with respect to economic losses, "most 
natural disasters are not random acts, but rather the direct 
and predictable consequence of inappropriate land use." 

While these examples share the four basic features of the 
HOT state, it can be difficult to precisely characterize and 
quantify the role of design in complex technical and bio- 
logical systems without going into great detail. Thus in 
any system there may be confusion as to which feature are 
due to design and which are due to dynamical or statis- 
tical mechanisms more familiar in physics. In advanced 
systems, designed features are so dominant and pervasive 
that we often take them for granted. While generic com- 
plexity emerges from a featureless substrate, the complexity 
in designed systems often leads to apparently simple, pre- 
dictable, robust behavior. As a result, designed complexity 
becomes increasingly hidden, so that its role in determining 
the sensitivities of the system tends to be underestimated 
by nonexperts, even those scientifically trained. Further- 
more, because HOT systems are simultaneously robust and 
sensitive to their components and environment, it is difficult 
to predict a priori which details are important. 

Nonetheless, the special sensitivity of HOT systems can 
lead to methods for separating features associated with 
design from those which collectively and self-consistently 
emerge from internal processes. When the behavior of a sys- 
tem changes radically in response to small rearrangements, 
variations in the boundary conditions, or the replacement of 
highly nongeneric (measure zero) elements with more ran- 
dom versions, then, chances are, the system is HOT. Evi- 
dence for HOT states can even be found in familiar labora- 
tory experiments. For example, water flowing in structured 
(straight, smooth) pipes can be laminar to Reynolds num- 
bers of 10'^. For the same pressure drop, this results in 
flows which are much greater than for more generic (e.g. 
rough, twisting, turning) pipes, which become turbulent at 
Reynolds numbers below 10'^. However, "designed" pipes 
are hypersensitive to microscopic details, such as small con- 
centrations of polymers, wall roughness, or vibrations. 

Random analogs of familiar systems in engineering and 
biology are so obviously different from our daily experience 
that the comparison is almost absurd. A truly random traf- 
fic system would have no lanes, dividers, traffic laws, colli- 



sion avoidance or other control systems, and would exhibit a 
"phase transition" of sorts at very low densities compared 
to the standard operating conditions of our current high- 
ways. It has been the work of auto makers, civil engineers, 
and city planners to continue developing new methods to 
increase throughput without exceeding a maximum yield 
point by incorporating increasingly structured and sophis- 
ticated features into each subsystem, if not globally opti- 
mizing the design. Computer networks, power grids, and 
biological systems are even more highly structured, with 
hierarchies, protocols, and enormous amounts of feedback. 
The message we extract from our simple percolation model 
is that in these systems a detailed representation of the 
internal interactions studied in a generic setting may be 
much less accurate than a coarser representation coupled 
with better characterizations of nongeneric elements such as 
perturbations and boundary conditions. In the HOT state 
design influences the most basic properties, and therefore 
must be taken into account throughout modeling, analysis, 
and simulation. 
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